Skip to content

Run Laya decisions fully in-process on ONNX Runtime - #55

Merged
elbruno merged 9 commits into
mainfrom
elbruno-onnx-local-models-research
Sep 30, 2026
Merged

elbruno merged 9 commits into
mainfrom
elbruno-onnx-local-models-research

Conversation

@elbruno

@elbruno elbruno commented Sep 30, 2026 •

Copy link
Copy Markdown
Owner

Why

ElBruno.LocalLLMs.Decisions shipped requiring a running laya-serve Python process, reached over loopback HTTP through the ElBruno.AI.Jev SDK. That put a Python install, a second process and a port in the path of every decision — a poor fit for a library whose whole premise is local, in-process inference.

This replaces that transport with pure managed .NET: Microsoft.ML.OnnxRuntime for the forward pass and Microsoft.ML.Tokenizers for the input. No server, no Python, no network call at inference time.

LayaDecisionClient becomes LayaOnnxDecisionClient. The package was never published to NuGet, so no consumer is broken.

What made this non-trivial

The model scores the hidden state at each [MASK] marker, so a single extra or missing token shifts every later marker and silently changes the answer rather than failing. Four pieces had to be ported exactly:

Component Role
LayaTokenizer + ByteLevelPreTokenizer Rebuilds the checkpoint's ByteLevel BPE from its own tokenizer.json. BpeOptions.ByteLevel only does byte↔unicode mapping and does not pre-tokenize, so the GPT-2 regex split is supplied separately — without it leading spaces vanish. HuggingFace also matches added tokens before BPE runs, including [MASK]'s lstrip, so that pre-pass is ported too.
LayaSequenceBuilder Ports build_sequence / render_options, preserving Laya's budget order: options, then instructions, then the state.
LayaRuntimeConfig Reads the fitted temperatures and clamps them into Laya's own 0.5–5.0 range.
DecisionMath Temperature divide → max-shifted softmax, so a large logit over a small temperature cannot overflow to NaN.

The tokenizer was the main risk. It was iterated to 22/22 exact parity against a transformers.js oracle covering Laya prompt fragments, Unicode, emoji, CJK, Cyrillic, whitespace runs and a literal [MASK].

Judgement call worth reviewing

Laya's English checkpoint ships choice:11+ = 0.1006, far below its own TEMP_MIN of 0.5. That sharpens the logits roughly tenfold. This clamps it into Laya's valid range and reports it per answer via CalibrationClamped on ChoiceResult, ScoreResult and ProbabilityResult — per answer rather than model-wide, because a checkpoint can mix valid and invalid buckets.

The clamp is a guardrail, not a fix. Measured on the real checkpoint, a 12-option choice still reports 100.0% confidence after clamping, because any temperature below 1.0 still sharpens. So the ranking stays reliable while the magnitude does not, and ChoiceOrNull will never abstain in that bucket. The guide and the new DecisionCalibration sample both say this outright.

Verification

Every path was exercised against the real 800 MB model, not just compiled:

  • Exact parity through the public API: billing 0.9696 / technical 0.0152 / sales 0.0152, matching the reference checkpoint's published model card.
  • Consumed as a real .nupkg, not a project reference — proving ONNX Runtime's native assets flow to consumers.
  • Cold start from an empty cache: downloads from HuggingFace and infers correctly in 93 s, including the model.onnx.data external weights the graph is unusable without.
  • Cache-hit path resolves in 3.5 s with no download.
  • Batching ≥ 2 confirmed (a known ONNX export hazard). Batched vs. single answers differ by roughly 1e-5 to 1e-4 depending on input and batch size, consistent with fp16 accumulation rather than marker misalignment. The DecisionCalibration sample prints this live.
  • Sample routes all three tickets correctly, four batched questions each, ~600 ms warm.
  • 79 package tests pass; full solution green apart from one pre-existing WSL-availability failure in MagenticUIServer (confirmed failing on a clean tree without these changes).

API change

DecisionOptions drops Endpoint / ApiKey / Timeout / Model in favour of ModelRepository, ModelPath, CacheDirectory and IntraOpNumThreads.

Laya's authors publish PyTorch weights only, so every ONNX export of Laya is community-produced. The default is now elbruno/laya-onnx, an unmodified mirror of inferenceprince/laya-onnx published so this library does not depend on a third party's repo staying available. It was verified identical two ways: every file matches by git/LFS OID, and inference on identical inputs differs by 0.00E+000. The source stays fully configurable so consumers can pin an export they control.

Docs, README, samples and CHANGELOG updated; version bumped to 0.22.0. Adds a DecisionCalibration sample that demonstrates the clamp and the fp16 batch variance with live numbers.

Bruno Capuano and others added 9 commits September 30, 2026 11:38
.NET 8 reaches end of support in November 2026, so the net8.0;net10.0
multi-targeting is dropped ahead of it. Every library, test, sample and
benchmark project now single-targets net10.0.

This is a prerequisite for consuming ElBruno.AI.Jev (a .NET 10 only
package) when adding local Laya decision support, since CI previously
forced the whole solution to build as net8.0.

- csproj: net8.0 / net8.0;net10.0 -> net10.0 (netstandard2.0 native
  packaging projects unchanged)
- global.json: SDK 10.0.0
- ci.yml / publish.yml / publish-bitnet-native.yml: install 10.0.x only
  and drop the per-framework overrides
- PlaceholderIntegrationTests: drop the hardcoded
  -p:TargetFrameworks=net8.0 from its dotnet pack call, which re-restored
  the library obj/ as net8.0 and broke every later net10.0 build
- docs, scripts and conventions updated to match

Verified: dotnet build (0 errors) and dotnet test (1953 passed). The one
remaining failure, MagenticUIServer.Agents CodeExecutorTool WSL test, is
environmental (local WSL mounts read-only) and unrelated; it early-returns
on the Linux CI runner.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: c71e4170-a73a-4ac2-8c59-4b80834e7968
Adds a new package that runs a local decision model returning typed
choice/score/yes-no answers with full probability distributions in a
single forward pass, rather than generating text.

Backed by Laya (Apache-2.0) over the Jev System One wire protocol,
reached through ElBruno.AI.Jev 0.6.0. The wire contract was validated
end to end against a live laya-serve instance: all three answer types
round trip, the request omits `model` so Laya auto-routes, and extra
Laya fields survive in RawRepresentation.

API:
- IDecisionClient with ChooseAsync / ScoreAsync / AskAsync, plus
  EvaluateAsync for batching several questions into one forward pass.
- DecisionRequest builder (immutable) and DecisionResult reader.
- ChoiceResult.ChoiceOrNull(threshold) so ambiguous inputs can fall
  through to a human instead of being confidently misrouted.
- ProbabilityResult exposes the probability, not just a boolean, and
  AtThreshold re-thresholds without re-running inference.
- AddLocalDecisions() DI registration, and an IJevDecisionClient
  constructor overload for testing without a server.

DecisionOptions.Endpoint is restricted to loopback so prompts cannot
leave the machine through a configuration mistake.

Laya's public checkpoints report uncalibrated confidence, and Laya
itself warns about out-of-range temperatures at startup. The guide,
README and sample all state plainly that DecisionThreshold must be
fitted on the user's own labelled data.

Also registers the package in Directory.Build.props sibling versioning
and adds its pack line to publish.yml.

46 new tests. Full solution: 1998 passing.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: c71e4170-a73a-4ac2-8c59-4b80834e7968
The Decisions package required a running `laya-serve` Python process reached
over loopback HTTP through the ElBruno.AI.Jev SDK. That put a Python install,
a second process and a port in the path of every decision, which is a poor fit
for a library whose whole premise is local in-process inference.

Replace the transport with pure managed .NET: Microsoft.ML.OnnxRuntime for the
forward pass and Microsoft.ML.Tokenizers for the input. `LayaDecisionClient`
becomes `LayaOnnxDecisionClient`. The package was never published, so no
consumer is broken.

Reproducing Laya's numbers required porting four pieces exactly, since the
model scores the hidden state at each `[MASK]` marker and a single extra or
missing token shifts every later marker and silently changes the answer:

- `LayaTokenizer` rebuilds the checkpoint's ByteLevel BPE from its own
  `tokenizer.json`. `BpeOptions.ByteLevel` only does byte<->unicode mapping and
  does not pre-tokenize, so `ByteLevelPreTokenizer` supplies the GPT-2 split;
  without it leading spaces vanish. HuggingFace also matches added tokens
  before BPE runs, including `[MASK]`'s `lstrip`, so that pre-pass is ported
  too. Validated at 22/22 against transformers.js.
- `LayaSequenceBuilder` ports `build_sequence` and `render_options`, keeping
  Laya's budget order: options, then instructions, then the state.
- `LayaRuntimeConfig` reads the fitted temperatures and clamps them into
  Laya's own 0.5-5.0 range. The English checkpoint ships `choice:11+ = 0.1006`,
  which sharpens the logits roughly tenfold and would report a 24% answer as
  99% confident.
- `DecisionMath` divides by temperature then softmaxes, shifting by the maximum
  so a large logit over a small temperature cannot overflow to NaN.

Verified end-to-end: `billing 0.9696 / technical 0.0152 / sales 0.0152`,
matching the reference checkpoint's published model card exactly, and the
sample routes all three tickets correctly with four batched questions each.

DecisionOptions drops Endpoint/ApiKey/Timeout/Model for ModelRepository,
ModelPath, CacheDirectory and IntraOpNumThreads, so the community ONNX export
used is fully configurable.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: c71e4170-a73a-4ac2-8c59-4b80834e7968
The default model source was inferenceprince/laya-onnx, a third-party
repository that could be renamed or deleted out from under consumers.
elbruno/laya-onnx now mirrors that export, verified byte-identical: all
six files match by git/LFS OID, and inference over identical inputs
produces a zero delta against the original.

The weights are unmodified fp16 and Apache-2.0, with attribution to the
Laya authors and the original exporter recorded in the model card.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: c71e4170-a73a-4ac2-8c59-4b80834e7968
squad-ci.yml shipped as an unfilled scaffold: it echoed "No build commands
configured" and exited 0 in four seconds, so it reported a passing status
check while testing nothing. It fired on PRs to main, which meant every PR
showed a green check that carried no signal.

It now runs the same restore/build/test the CI Build workflow runs,
including -p:NoBuild=false, which the BitNet native projects need or they
are silently skipped.

main is dropped from its triggers because ci.yml already covers main and
running the suite twice on one commit buys nothing. dev, preview and
insider do not exist yet, so this workflow is dormant until the dev-first
branches are created — at which point they get real CI instead of none.

Validated with actionlint.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: c71e4170-a73a-4ac2-8c59-4b80834e7968
Both described the weights as simply coming from HuggingFace without
saying which repository, so neither reflected the move to the
owner-controlled mirror.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: c71e4170-a73a-4ac2-8c59-4b80834e7968
…veats

The choice:11+ temperature clamp was tracked with an internal, model-wide flag
that callers could not read. A checkpoint can mix valid and invalid buckets, so
the actionable question is whether a given answer's own bucket was clamped.

Track clamping per temperature entry and surface it as CalibrationClamped on
ChoiceResult, ScoreResult and ProbabilityResult.

Running this against the English checkpoint shows the clamp is a guardrail, not
a fix: a 12-option choice still reports 100.0% confidence, because clamping to
0.5 bounds the tenfold sharpening but any temperature below 1.0 still sharpens.
The docs previously claimed clamping "keeps the number honest", which is wrong.
Correct that, and state plainly that ranking is reliable while the magnitude is
not.

Also document fp16 batch variance, measured at 1.2e-05 to 1.6e-05 between a
question asked alone and the same question asked inside a batch of four.

Add a DecisionCalibration sample that demonstrates both effects with live
numbers, and register it in the solution.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: c71e4170-a73a-4ac2-8c59-4b80834e7968
The Pack step packs ElBruno.LocalLLMs.Decisions with --no-build, but the Build
step never built it and nothing else references it, so the Release output never
existed. Reproduced against a clean output directory: pack fails with NU5026,
'ElBruno.LocalLLMs.Decisions.dll to be packed was not found on disk'.

Any release published from this workflow would have failed at the Pack step.

Build the project alongside the other packable ones and run its test project in
the Unit Tests step so the package cannot ship untested.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: c71e4170-a73a-4ac2-8c59-4b80834e7968
…lish

Promote the live Unreleased section to 0.22.0, matching the existing dated
heading convention.

Also run ElBruno.LocalLLMs.BlazorComponents.Tests in the publish workflow. The
package was packed and pushed without its 23 tests ever running in the release
gate. Verified passing under the same Release configuration the gate uses.

Two stale Unreleased sections remain further down the file, describing features
that shipped long ago. They predate this work and are left untouched.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: c71e4170-a73a-4ac2-8c59-4b80834e7968
@elbruno
elbruno merged commit 934029a into main Sep 30, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant